Towards Evasive Attacks : Anomaly Detection Resistance Analysis on the Internet

TOWARDS EVASIVE ATTACKS: ANOMALY DETECTION RESISTANCE ANALYSIS ON THE INTERNET Jing Jin, PhD George Mason University, 2013 Dissertation Director: Dr. Jeff Offutt The Internet is rapidly improving as a platform for deploying sophisticated interactive applications especially in Web 2.0. Although the shift from desktop-centric applications brings many benefits to web-based computing and cloud computing, such as efficient communication with ubiquitous access and availability, the way that Internet users share and exchange information also opens their own information to security problems. Today, attackers conduct malicious activities to routinely track the identities of internet-connected users, steal privacy data, abuse users personal information, and expose the users unwanted data or programs. Although these attackers can also accomplish these goals by other means, the Internet has made it much easier for attackers to locate victims, discover sensitive information and initiate unsolicited communication with the victims. To detect attacks from the Internet, anomaly detection methods have been proposed to compare abnormal behavior from malicious activities with legitimate behavior. While detection techniques have been developed, evasive techniques have not been widely studied. This dissertation explores the limitation of current anomaly detection in the context of the battle between detectors and attackers by finding potential evasive attacks and measuring detection resistance of evasive techniques. This dissertation studies detection resistance at user application and IP layer. This dissertation first explores the limitations of current Human Observational Proofs (HOP) based bot detection systems by creating a new evasive bot system that masquerades as human beings on the Web. Specifically, I characterize the existing HOP-based web bot detectors and develop an evasion framework based on human behavior patterns. Instead of subverting a specific detection system, the major goal of this study is to provide a systematic approach to evaluate and explore the limitations of current HOP-based detection systems on the web. This dissertation also explores the limitations of IP timing covert channel detection systems by analyzing the stealthiness of timing covert channels. For evasive techniques, this dissertation studies passive detection resistance and active detection resistance with various evasive methods such as mimic, mix and replay, coding scheme rotation, etc. It defines a new measurement approach to evaluate covert channel evasion capabilities. The major goal of this study is to provide a systematic approach to better understand the design of IP timing covert channels. Both studies use similarity measurement that measures the similarity between legitimate behavior and abnormal behavior. This similarity measurement evaluates the capability of evasion against detection methods with detection independent approach.